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I. EXECUTIVE SUMMARY 



This paper describes an experiment based on 94,560 voice utterances 
which examined the stability of voice recognition performance over a 
period of 21 weeks. The purpose was to determine how consistently the 
voice reference patterns would work and thus indicate how much re-training 
might be needed over extended periods of time. Recognition performance as 
a function of vocabulary size up to 240 utterances was also examined. In 
addition, a small effort was undertaken to examine recognition performance 
when 2 people combined their training passes, i.e. if the voice recognition 
equipment required X training passes for each utterance, how well would 
the recognizer work if one person provided half of the utterances and 
another person provided the other half of the utterances? 

Eight subjects participated in the experiment in which the voice 
recognizer was tested by having each subject repeat 480 utterances to the 
recognizer each week for 21 weeks in a relatively noise free environment. 

Voice patterns were not allowed to be retrained or "fine tuned" 
during the study. 

In addition, a male and a female made a joint set of reference patterns 
with half the training passes for each utterance provided by the female and 
half by the male. 

Longitudinal results showed performance was statistically stable over 
the 21 weeks with no serious degradation occurring as time elapsed. For 
the 8 subjects who tested the recognizer against their own voice patterns, 
the recognition error rate over 21 weeks was 2.07% including both mis- 
recognitions and rejections. In terms of mis-recognitions only, the error 
rate was 1.65%. In addition, it should be noted that two people (1 male 
and 1 female) accounted for 57.37% of these errors. 



With respect to vocabulary size, analysis showed no statistical dif- 
ferences in performance when the vocabulary size was 20, 40, 60, 80, 100, 
120, 140, 160, 180, 200, 220 or 240 utterances. 

The two subjects who combined their voice reference patterns had an 
average error rate of 1.87% with only 6 rejects in total between them. 

This small investigation into the joint reference pattern concept 
suggests that if two persons are on a job and one is unable to continue, 
the other person could step in and start using the same voice patterns, 
even though the recognizer is a speaker dependent system. 

In summary, voice recognition performance was: 

1) Stable over time. 

2) Stable over vocabulary size. 

3) Quite good when two people combined their voice 
reference patterns in a speaker-dependent voice 
recognition system. 
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II. INTRODUCTION-OBJECTIVE 



This paper describes an experiment whose purpose was to examine voice 
recognition performance as a function of time and vocabulary size. In 
addition, a small sample was taken to examine the possible use of joint 
training patterns by two people. Previous research by others working for 
this writer is shown in Appendix A. The results of some of these studies 
suggested voice input was a potential way for improving job performance in 
various applications. Many of these studies were of a relatively short 
duration of several weeks and it was intriguing to know if the results 
would have held up over a longer period of time. Hence this study. 

III. SUBJECTS 

Six military officers and two civilians participated in the longitudinal 
study over 21 weeks. Two of the officers were female. All subjects were 
between 20 and 40 years old. Their experience with voice recognition 
equipment was limited to about a month's previous experience with voice 
recognition systems, except for 1 male and 1 female who had a year or more 
of experience. Military ranks ranged from Lieutenant to Lieutenant 
Commander in the Navy and Captain to Lieutenant Colonel in the Air Force. 

IV. INITIAL TRAINING 

The eight subjects individually trained a Threshold Technology, Inc., 
Model T600 voice recognition system on a vocabulary of 240 utterances. 

After the vocabulary was trained, each utterance was repeated three times 
with the vocabulary wide open with no branching or structuring. If the 
voice recognition system correctly recognized the utterance 2 out of 3 
times, the training pattern was considered valid... it was retrained if 
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it didn't meet this criteria and tested again until successful validation 
was acquired. The T600 initially requires 10 training passes which seems 
to provide quite good reference patterns. Less than 2% of the initially 
trained utterances even needed to be retrained to meet the validation 
criteria. 

Upon successful training, each subject's voice patterns were recorded 
on individual tape cassettes using the tape recorder of the T600. The 
training patterns were not allowed to be altered for the next 20 weeks. 

In addition, one female and one male (both with the most experience 
with voice recognition equipment) also trained the recognizer in a joint 
mode on all 240 utterances in which 5 training passes for each utterance 
were provided by the female and 5 by the male. The validity of these 
training passes also had to pass the 2 out of 3 criteria (mentioned 
earlier) for both the male and female. The joint reference patterns were 
then recorded on a tape which was not altered for the next 20 weeks. 

V. THE VOCABULARY 

The vocabulary of 240 utterances is shown on the data sheet in 
Appendix B. It is divided into groups of 20. Each group of 20 was 
balanced with two 1 syllable utterances, six 2 syllable utterances, four 
3 syllable utterances, four 4 syllable utterances and four utterances of 
5 or more syllables. These were selected to attempt to simulate their 
frequency of use in a command center. 

VI. PROCEDURE AND DATA COLLECTED 

Each week for 20 weeks, 7 of the subjects were observed and measured 
on recognition performance as they repeated the 240 utterances twice per the 
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instructions. In the T600, one can set a given utterance's prompt to null 
and deactivate that utterance and any beyond it. 

Therefore, each week a subject's voice pattern tape was read into 
the recognizer memory and immediately prompt 20 was set to null so only 
voice reference patterns (p through 19 were active. Then the subject 
repeated these utterances once and then went through them again. The first 
20 utterances were therefore being tested against themselves. Next prompt 
# 20 was reset and prompt # 40 was set to null so now reference patterns (}> 
through 39 were active. Then the subject repeated the second 20 utterances 
twice, but in this case the second set of 20 utterances were actually being 
tested against all 40 active voice reference patterns. Then when the 
third set of 20 utterances were tested, they were actually tested against 
an active vocabulary of 60. Therefore, the vocabulary was opened up by 
groups of 20 until the last 20 utterances were being tested against an 
active vocabulary of 240 reference patterns. 

Each subject, except one who could only participate for 8 weeks, 
provided 480 utterances per week for 21 weeks. All eight subjects therefore 
contributed a total of 74,400 tests of the recognizer's capability. 

In addition, the male and female who had made the joint reference 
pattern tape did "double" duty by each providing another 480 utterances 
per week which were tested against the patterns on the tape which had 
their joint reference patterns. 

These two subjects contributed 20,160 utterances in this portion of 
the experiment. 

After 20 weeks, all subjects retrained every utterance which had a 
misrecognition at any time during the first 20 weeks. After validation 
criteria were again satisfied, data for the 21st week was collected. 
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The subjects with the joint tape did likewise. Every utterance 
which previously had a misrecognition by either the male or female was 
retrained therefore by both subjects. After validation, data for the 21st 
week was collected. 

All data was collected in a relatively quiet atmosphere typical of 
normal operations in a command center environment or office. 

VII. RESULTS 

Results for the longitudinal portion of the research are shown in 
Figure 1. An analysis of variance on the arcsin transformation of the 
raw data over each week by each subject is shown in Table 1. Subject 4 
was not included in the statistical analysis due to his participating in 
the study for only 8 weeks. The statistical results of Table 1 showed an 
expected difference between subjects but no difference in performance by 
week over the 21 weeks. This is seen by observing the overall average 
curve (the middle one) in Figure 1 which shows less than a 1.7% variation 
from week to week over 21 weeks. Although the averages for the males and 
females are shown in this figure individually, the middle average line is 
more indicative of the group's performance since the females' line is 
based on 2 subjects whereas the males' is based on six subjects. The 
overall variability from week to week was large enough to show no statistical 
difference in general over the 21 weeks of the study. As can be seen, per- 
formance did not get worse and worse over time and thus the initial voice 
reference patterns provided stable performance for the 21 weeks. 

This indicates that large amounts of retraining may not be required 
over extended periods of time, and that performance can remain consistent. 

The reader will recall that after 20 weeks, all utterances which 
previously had any misrecognitions were retrained. One might have thought 
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TABLE I 



Analysis of Variance on 
Subjects over 21 Weeks 



Source of 



Variation 


df 


MS 


F 


SUBJECTS 


6 


.4130 


82.60* 


WEEKS 


20 


.0056 


1.12 


RESIDUAL ERROR 


120 


.0050 




TOTAL 


146 






* 








p < .01 








N.S. = not significant 
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Figure 1= Error rote versus weeks 
( Individual patterns) 
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this would help and subsequently cause a large improvement in performance 
during the 21st week. However, as seen in Figure 1, there was a minimal 
improvement but nothing statistically significant in the 21st week. 

One will notice the average error rate did increase slightly in 
week 8. This may have no meaning or could be related to exams which would 
have been given about that time. Also week 14 showed a fair drop in error 
rate and may have been related to the fact that this was at the beginning 
of a new academic quarter in which subjects may have been under less 
pressure. Since these observations are being made after the experiment, 
one can not take the above implications as true fact. However, it is 
something to consider and is a small indication of possible stress factors 
at work. It helps to point out that much remains to be done in the whole 
area of environmental and psychological stress effects on voice recognition 
performance. 

Figure 2, shows the results of the investigation into vacabulary size. 

It can be seen that in general, voice recognition performance remained 
relatively constant as the size of the vocabulary increased. Statistical 
analysis showed that the error rate did not increase as a function of 
vocabulary size as many might expect. Although this writer claims no 
magic credit for the following, a lot of time was spent developing the 
vocabulary used here which may be typical of a vocabulary used in command 
center operations. In addition, each group of 20 utterances was balanced 
with equal numbers of utterances of 1,2, 3, 4 and 5 (plus) syllable words. 

The combination of vocabulary choice and syllable balancing may contribute 
to performance stability as a function of vocabulary size. 

As a result of using different numbers of syllables in the 
vocabulary, it was also possible to get an indication of how well utterances 
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Figure 2' Mean error rate versus vocabulary size. 
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with different numbers of syllables were recognized. This is shown in 
Figure 3 which illustrates a declining error rate as a function of the 
number of syllables in the utterance. 

Figure 4 shows the results of the performance for each subject over 
all 21 weeks. Subjects 1 (male) and 6 (female) were the participants 
in the joint reference pattern investigation. For both these subjects, 
their performance dropped about .7% when using the joint patterns versus 
using their own patterns. It is also interesting to note that subject 1, 
using joint patterns of female subject 6, performed better than any other 
subject did against their own individual reference patterns. Subject 4 
in this figure is the one who only participated 8 weeks. 

Figure 5,6, and 7 simply show the individual subject's error rates 
by week for the entire study. Subject 1 and 6 in Figure 5 are the same 
subjects as in Figure 7 but using their individual voice reference 
patterns versus their combined patterns. 
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MEAN ERROR RATE (%) 
(rejects + misrecognitions) 




Figure 3' Error rate versus syllobles 
(over oil conditions) 
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[pouD) SUBJECTS (po?rtrn») 



Figure 4‘ Recognition performance by subject on individual patterns 
unless noted. 
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Figure 5- Performoncc by week for Individual subjects 
(own voice pottern). 
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Figure 6^ Performance by week for individual subjects 
(own voice patterns). 
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Figure 7 •• Performance by week for two subjects using 
combined voice reference patterns. 
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VIII. CONCLUSIONS 



The main purpose of this study was to determine if voice recognition 
performance could be stable over a "long" period of time. It is apparent 
that relative stability is possible and one may not have to expect to do 
a lot of re-training of voice patterns as time progresses over many months. 
As a matter of record, this writer uses two year old voice reference 
patterns on a daily basis and hardly ever re-trains any of the old patterns. 

As a result of the main experiment, it was also possible to get an 
idea of recognition performance as a function of vocabulary size. 

Although one might believe that the error rate should increase dramatically 
as larger vocabularies are used, the current results indicate that this 
need not be the case. Rather, it might be argued, one can overcome large 
vocabulary difficulties through judicious selection of the vocabulary 
items themselves, as others have suggested in past discussions at con- 
ferences, etc. 

Finally, it was successfully shown that a male and a remale could 
combine their voice reference patterns in a speaker-dependent recognition 
system and still achieve high recognition performance. This has many 
implications for use in emergency situations where one member of a crew 
may become incapacitated suddenly. If one didn't wish to have joint 
patterns for all utterances as done in this study, one might at least 
have joint patterns for emergency and "STOP-ACTION" types of commands in 
the vocabulary. 
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APPENDIX A 



VOICE STUDIES AT NPS 



This project is one of several voice recognition research projects 

conducted for/by Professor G. K. Poock at NPS over the last several years. 

The complete list, in addition to this report, includes: 

Armstrong, J. W. , The Effects Of Concurrent Motor Tasking On Performance 
Of A Voice Recognition System , Masters Thesis, Naval Postgraduate School, 
Monterey, 1980. 

Batchellor, M. P. , Investigation Of Parameters Affecting Voice Recognition 
Systems In C^ Systems, Masters Thesis, Naval Postgraduate School, Monterey, 
1981. 

Bragaw, P. H. , Investigation Of Voice Input For Constructing Joint Chief s 
Of Staff Emergency Action Messages , Masters Thesis, Naval Postgraduate 
School, Monterey, 1981. 

Jay, G. T. , An Experiment In Voice Data Entry for Imagery Intelligence 
Reporting , Masters Thesis, Naval Postgraduate School, Monterey, 1981. 

McSorley, W. J. Using Voice Recognition Equipment To Run The Warfare 
Environmental Simulator (HES), Masters Thesis, Naval Postgraduate 
School, Monterey, 1981. 

Naval Postgraduate School Report NPS54-80-010, The Effects Of Certain 
Background Noises On The Performance Of A Voice Recognition System , 
by R. Elster, September 1980. 

Naval Postgraduate School Report NPS55-80-016, Experiments With Voice 
Input For Command And Control: Using Voice Input To Operate A Distributed 

Computer Network , by G. K. Poock, April 1980. 

Naval Postgraduate School Report NPS55-81-003, Examination Of Voice 
Recognition System To Function In A Bilingual Mode , by D. E. Neil and 
T. Andreason, February 1981. 

Taggart, J. L. and Wolfe, C. D. , Speech Recognition As An Input Medium For 
Preflight In The P3C Aircraft , Masters Thesis, Naval Postgraduate School, 
Monterey, 1981. 



18 



APPENDIX B 



DATA SHEET 



NAME 



LONGITUDINAL STUDY 



DATE 



Is this data based on: 



I I Your voice patterns? 

I I Yours and other’s patterns? 

(\^10SG?) 

I I Other’s patterns? 

(Whose?) 



There are 240 patterns on your tape (word number 0 - 239 in memory). 
INSTRUCTIONS: 

A. Set the prompt of word 20 to a carriage return. 

B. Say word 21 to check your vocabula^size (it shouldn’t work). 

C. Say utterances 0 through 19. 

D. Then repeat utterances 0 through 19. We are getting 2 obser- 
vations per utterance. 

E. If the machine beeps, write ’’BEEP” to the right of the word number. 

F. If the machine makes a mistake, write the MISTAKEN output to the 

right of the word number. 

Now go through the list twice on the next page. 
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WORD # 


UTTERANCE 


0 


ONE 


1 


TWO 


2 


YANKEE 


3 


AIR ROUTES 


4 


GARY POOCK 


5 


LOAD THE GANN 


6 


CARRIAGE RETURN 


7 


LOAD THE SERVER 


8 


IRAN 


9 


JAPAN 


10 


SWEDEN 


11 


EUROPE 


12 


LOGIN POOCK 


13 


LEVEL TWO 


14 


ACCAT TITLE 


15 


STRAIT OF HORMUZ 


16 


LOAD GLD3 


17 


CONNECT TO CHARLIE 


18 


POOCK NPS PASSWORD 


19 


CHANGE DIRECTORY TO HUNTER 
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Now SET the prompt of word 20 back to **three*' and the prompt of word 40 
to carriage return. Check vocabulary size by saying word 41. It should 
not work. Now go through the following list twice. 



20 


THREE 


21 


FOUR 


22 


LOGOUT 


23 


GRAPHICS 

0 


24 


RED SPHERE 


25 


STEAM PLANT 


26 


ZERO 


27 


SEVEN 


28 


NOVEMBER 


29 


MOVE IT DOWN 


30 


USE THAT ONE 


31 


SPIROGRAPH 


32 


CAPTAIN EBBERT 


33 


CLOSE OUT CHARLIE 


34 


UP IN DETAIL 


35 


UNITED STATES 


36 


LEVEL TWO VIEWER 


37 


NORTH ATLAJ^TIC MAP 


38 


GENISCO ZERO PARAMETERS 


39 


MEDITERRANEAN MAP 
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Now set the prompt of word 40 back to ’’FIVE" and set the prompt of word 
60 to carriage return. CHECK vocabulary size by saying word 61. It 
should not work. Now go through the following list twice. 



40 


FIVE 


41 


SIX 


42 


ALPHA 


43 


BRAVO 


44 


0 

CHARLIE 


45 


DELTA 


46 


ECHO 


47 


FOXTROT 


48 


JULIETT 


49 


ROMEO 


50 


MOVE IT LEFT 


51 


SIERRA 


52 


SAN FRANCISCO 


53 


APPLICATION 


54 


ENGINEERING 


55 


HUMAN FACTORS 


56 


VOICE TECHNOLOGY 


57 


CENTRAL EXPRESSWAY 


58 


RUSSIAN VERSION OF HORMUZ 


59 


FILE TRANSFER PROTOCOL 
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Now Now set the prompt of word 60 back to "EIGHT" and set the prompt 



of word 


80 to carriage return. Check vocabulary size by saying word 


81. It 


should not work. Now go through the following list twice. 



60 


EIGHT 


61 


NINE 


62 


HOTEL 


63 


INDIA 


64 


KILO 


65 


LIMA 


66 


OSCAR 


67 


POPPA 


68 


MOVE IT RIGHT 


69 


UNIFORM 


70 


VIETNAM 


71 


KOREA 


72 


ADVISORY 


73 


INTERACTIVE 


74 


BUSINESS MEETING 


75 


CONTINUOUS 


76 


SPEECH RECOGNITION 


77 


CONTINUOUS SPEECH 


78 


EFFICIENT TRANSMISSION 


79 


SYSTEM INTEGRATION 
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Now set the prompt of word 80 back to "GOLF" and set the prompt of word 
100 to carriage return. Check vocabulary size by saying word 101. It 
should not work. Now go through the following list twice. 



80 GOLF 

81 MIKE 

82 QUEBEC 

83 TANGO 

84 VICTOR 

85 WHISKEY 

86 XRAY 

87 ZULU 

88 MOVE IT UP 

89 BANGLADESH 

90 TOKYO 

91 HOLLISTER 

92 DOWN IN DETAIL 

93 CORPORATION 

94 CRITERIA 

95 ADVANTAGES 

96 SUITABILITY 

97 RADIOLOGY 

98 IDENTIFICATION 

99 AUTOMATIC RECOGNITION 
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Now set the prompt or word 100 back to "COURSE'* and set the prompt of 
word 120 to carriage return. Check vocabulary size by saying word 121 
It should not work. Now go through the following list twice. 



100 


COURSE 


101 


SPEED 


102 


COMMAND 


103 


ATTACK 


104 


BINGO 


105 


REPORT 


106 


PROCEED 


107 


STATION 


108 


ALTITUDE 


109 


RECOVER 


110 


RELOCATE 


111 


DESIGNATE 


112 


AVAILABLE 


113 


PLOT ESM 


114 


TRACK ENEMY 


115 


DESIGNATE TRACK 


116 


COMMAND AND CONTROL 


117 


PROBABILITY 


118 


ENEMY DETECTION 


119 


PROBABILITY OF DETECTION 
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Now set the prompt of word 120 back to **LAUNCH^' and set the prompt of 
word 140 to carriage return. Check vocabulary size by saying word 141. 
It should not work. Now go through the following list twice. 



120 


LAUNCH 




121 


FIRE 




122 


CANCEL 




123 


MESSAGE 




124 


BEARING 




125 


LABEL 




126 


ORDERS 




127 


COPY 




128 


SATELLITE 




129 


ENVELOPE 




130 


NEGATIVE 




131 


CORRELATE 




132 


COMBINATION 




133 


SENSOR DELAY 




134 


MANEUVER DELAY 




135 


ALABAMA 




136 


TASK FORCE COMMANDER 


137 


NORTH CAROLINA 




138 


PROCEED TO NEW 


DELHI 


139 


PLACE A CIRCLE 


ON MOSCOW 
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Now set the prompt of word 140 back to "TIME" and set the prompt of word 
160 to carriage return. Check vocabulary size by saying word 161. It 
should not work. Now go through the following list twice. 



140 


TIME 




141 


SHOOT 




142 


SURFACE 




143 


REFUEL 




144 


MINEFIELD 




145 


DISTANCE 




146 


SHORE BASED 




147 


CONTACT 




148 


EXECUTE 




149 


SUBMARINE 




150 


ENEMY 




151 


ORDER NAME 




152 


CONNECTICUT 




153 


INDIANA 




154 


OKLAHOMA 




155 


PENNSYLVANIA 




156 


CALIFORNIA 




157 


SOUTH DAKOTA 




158 


PLACE A MARKER 


ON PARIS 


159 


BINGO ALL CRAFT 


IMMEDIATELY 
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Now set the prompt of word 160 back to ’’MAP” and set the prompt of 
word 180 to carriage return. Check vocabulary size by saying word 181 
It should not work. Now go through the following list twice. 



160 


MAP 


161 


GRID 


162 


NEUTRAL 


163 


MISSILE 


164 


SENSOR 


165 


ADAK 


166 


STOCKTON 


167 


NEW YORK 


168 


AIR FIELD NAME 


169 


TRACK UNKNOWN 


170 


TRACK FRIENDLY 


171 


TRACK NEUTRAL 


172 


BEARING AND DISTANCE 


173 


LOUISIANA 


174 


MINNESOTA 


175 


COLORADO 


176 


EISENHOWER 


177 


NEW MEXICO 


178 


RELOCATE THE SUNFISH 


179 


REFUEL THE CONNIE 
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Now set the prompt of word 180 back to **TAKE** and set the prompt of 
word 200 to carriage return. Check vocabulary size by saying word 201 
It should not work. Now go through the following list twice. 



180 


TAKE 


181 


PLACE 


182 


GEORGIA 


183 


VERMONT 


184 


TEXAS 


185 


DANIELS 


186 


UTAH 


187 


PLATFORM 


188 


LATITUDE 


189 


LONGITUDE 


190 


OHIO 


191 


TORPEDO 


192 


FLIGHT CONTROLLER 


193 


TRANS WORLD AIRLINES 


194 


PANGO PANGO 


195 


KEEP ON STATION 


196 


LAY A BARRIER 


197 


GROUND CONTROL APPROACH 


198 


ATTACK BARRIER TARGET 


199 


ATLANTIC DATA BASE 



29 



Now set the prompt of word 200 back to "SCOPE** and set the prompt of 
word 220 to carriage return* Check vocabulary size by saying word 221 
It should not work. Now go through the following list twice. 



200 


SCOPE 


201 


DROP 


202 


BANGKOK 


203 


BOMBAY 


204 


BRISBANE 


205 


CANTON 


206 


ANTWERP 


207 


AFRICA 


208 


ARKANSAS 


209 


SAIGON 


210 


USER'S GUIDE 


211 


KITTY HAWK 


212 


ACAPULCO 


213 


VLADIVOSTOK 


214 


YOKOHAMA 


215 


SEA OF JAPAN 


216 


DIEGO GARCIA 


217 


INDONESIA 


218 


PACIFIC DATA BASE 


219 


ARABIAN TANKER 
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Now set the prompt of word 220 back to "MAINE" Now go through the 
following list twice. 



220 


MAINE 


221 


SAVE 


222 


PORTLAND 


223 


RANGOON 


224 


ASPRO 


225 


KIEV 


226 


RED FOX 


227 


NAPLES 


228 


BLUE FORCE ONE 


229 


CALCUTTA 


230 


BALTIMORE 


231 


WYOMING 


232 


SEVASTOPOL 


233 


HONOLULU 


234 


KRENOMETR 


235 


JOHN KENNEDY 


236 


PLOT ALL SUBMARINES 


237 


UNITED AIR LINES 


238 


IBERIAN CARRIER 


239 


WEST GERMAN TORPEDO 



Now RETURN this data sheet to GARY POOCK. 
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